Virtual reality may seem like a new technology, but forms of immersive experience have a long history during which scientific and imaginative aspects often developed hand in hand. Host Amanda Anderson talks with a humanities scholar (Massimo Riva) and a cognitive scientist (Fulvio Domini) about their collaborative teaching project on the history and science of virtual reality, part of a larger initiative at Brown University in the collaborative humanities.

Transcript | Spring 2020 Student Projects Video

From the Cogut Institute for the Humanities at Brown University, this is Meeting Street. In this episode, I will speak with two Brown University faculty, one from the humanities and one from cognitive science, who partnered to teach an innovative seminar on the history and science of virtual reality. The course was offered as part of the Cogut Institute’s Collaborative Humanities program, which seeks to build research-based partnerships across the university. We will hear today about an extraordinary experience of cross-disciplinary pedagogy.

Let me introduce my guests. Fulvio Domini is a professor in the Department of Cognitive, Linguistic and Psychological Sciences. His research team investigates how the human visual system processes 3D visual information to allow successful interactions with the environment. His approach is to combine computational methods and behavioral studies to understand the visual features that establish the mapping between vision and action. His work is currently supported by a grant from the National Science Foundation.

My second guest, Massimo Riva, is professor and chair of Italian Studies. His research focuses on modern visual culture from a digital humanities perspective. His work has been recognized with three major grants from the National Endowment for the Humanities and a digital innovation fellowship from the American Association of Learned Societies. He has also recently completed a digital monograph, a pilot project of the Brown Digital Publications Initiative, supported by the Andrew W. Mellon Foundation. It is entitled “Italian Shadows: A (Curious) History of Virtual Reality.”

Massimo and Fulvio, welcome to Meeting Street. So let’s begin our conversation. Your course, Simulating Reality, explores the history and science of virtual reality, or immersive experiences more generally. Fulvio, maybe we can start with you. From your perspective, what is gained from combining these two disciplinary approaches?

Well, first, thank you for inviting us to this interview. I think it’s a great initiative. So, to answer your question is, you know, when we start from a scientific point of view, VR is a means to reproduce reality, OK? But an underlying assumption, that’s when we talk about reality, we intend a specific reality, which is the physical reality, the reality we describe through physics and geometry and so forth. And so this is the approach we take when we study the brain. So we study the brain from the study [of] how the sensory signals — so our vision, our touch, the sound, the taste, the smell, and so forth — are transformed back into the reality they came from, OK? So that’s the approach.

But the big question that I think is still being asked by philosophers, by scientists, and so forth, even in the physical sciences: What is reality? So is a physical description of the world reality? What about our subjective experiences? What about our emotional response to an image or an immersive experience? The experience that we get in a virtual reality environment. Also, the experience we get by just looking at the picture or looking at the flower and so forth. So within all these experiences what is also embedded is the cultural and societal context that influences these experiences. And so by looking at art and history and the humanities in general, we can really gain a valuable insight to answer these important questions.

Thank you. Massimo, how about from your perspective? What was gained by partnering with Fulvio?

First of all, thank you for inviting us, and I think Fulvio really described very well also the reasons why a scientific perspective is crucial. One of the central topics, if not the central topic, of our seminar is simulation, is simulating reality. And simulation has many different meanings, from a scientific point of view, from the humanistic point of view.

And so I think that gaining a better understanding and grounding our understanding into physical processes is a challenge and also a fundamental, you know, benefit for a humanist trying to understand all the different nuances of our constant, continuous, and always-renewed simulation of reality through the arts, through language, through many of our gestures, and so on and so forth. So, a comprehensive understanding of this phenomena is perhaps what we both aim at from our different angles.

Massimo, what has changed historically in the approach to immersive experiences? Are there important turning points or breaks in the history over the course of the modern period to begin with? Or alternatively, is there an important through line?

Well, I would say that both perspectives are true. On the one hand, scholars have been talking about the scopic regimes or modes of vision — since in our course we consider primarily vision — and have established how these modes of vision are embedded in or actualized by technological apparatuses. So, thus linking these changes of paradigm or epistemological breaks in the way vision is conceived, how we look at things and what we actually see, to the history of technology and science.

So discoveries and inventions of telescopes, microscopes, etc. literally changed our vision of the world, our perception of reality, and generated new theories about reality. On the other hand, there is also historical continuity, characterized by the centrality of vision for our cognition and experience of reality, established by Western classical thought. In fact, this dialectic between paradigm changes and historical continuity shapes the multiple meanings and the psychological effect of simulation. A good example is the history of the camera obscura, from Aristotle and al-Hassan to Leonardo da Vinci and Giambattista della Porta all the way up to the invention of photography and cinematography in the 19th century.

So the camera obscura is fundamentally a box or a room into which we can project a virtual image of the outside world through a simple hole or a system of lenses. And the physical principles that make it work are relatively simple. In the 17th century Kepler basically conceived of the camera obscura as the model of the human eye, constructing his theory of the retinal image.

Yet the effects of the camera obscura are also wondrous, amazing. Following Athanasius Kircher, Giambattista della Porta, also in the 17th century, turned it into a kind of illusionistic stage for magical tricks. So, thus the camera obscura illustrates both continuity and change. It can be used for both experimental and theatrical purposes, and this complicates the meaning of simulation as well.

You know, historically, over the past few centuries, we have gradually moved from a representational mode, virtualization of the real in artistic reproductions and representations, paintings, sculptures, theater, films, etc. to what we may call a thoroughly simulating mode. So a realization of the virtual. Thanks to digital technology, we are now able to produce techno-cultural experiences that are entirely artificial but feel more real than the real.

Massimo, how did your previous scholarship prepare you to teach this course?

So I have just completed, as you mentioned in the introduction, a digital monograph entitled “Italian Shadows: A (Curious) History of Virtual Reality,” part of the Brown Digital Publications Initiative. It will be published next year by Stanford University Press. So this project consisted in a series of case studies focused on analog optical devices and gadgets, such as the camera obscura, the magic lantern, the stereoscope, illustrating the cultural shift that I just mentioned from the virtualization of reality to the realization of the virtual. The most innovative component of this project is a series of digital simulations that both demonstrate how these devices actually work and provide an interpretation of their symbolic effects within their specific historical context in 18th-century Venice or late 19th-century America.

To give you an example, going back to the camera obscura: In the 18th century, the camera obscura was, above all, a device used by painters in order to create virtual reproductions of the world as accurate as possible. For example, Canaletto’s reproductions of views of Venice were widely marketed during the grand tour. This is one way of simulating reality.

Creating a copy of it as faithful as possible, a copy that possesses or communicates an aura even superior to our perception of the real thing, which is inevitably affected by environmental factors. One could compare Canaletto’s ideal views of Venice with the experience of being in Venice, negotiating the crowds, at least before COVID.

So the experience of being in Venice without having to go to Venice, to go there, was in fact used as an advertisement for the stereoscopic travel kits produced and marketed by American firms such as Underwood & Underwood in the late 19th and early 20th century, which is the last case study in my monograph. So I saw this course as a great opportunity to test my ideas and the methodology.

That’s fascinating. Fulvio, what about your own research and academic history do you think made you receptive to a collaboration with a humanities scholar?

Yeah, I mean, what Massimo just described, all these different dimensions, dimensions that we can consider when we talk about simulating reality, when we talk about art, and the history of art, I think all of these examples are what attract people like me who are doing more of a dry science, like our kind of science.

So what we do is — you know, what we’re trying to understand is how the human visual system from this pattern of rays, the fall of the eyes, can figure out where things are in the world, what the shape of these things are, how we can interact with those things.

What is the value of interacting with those things and so forth. So this is dry in the sense that at the end of the day, if you look at the history of our scientific approach, and also you look at the engineering part of it, those who tried to build machines that see in the same way that humans see, you realize that what all these approaches are concerned about is to reproduce with as much fidelity as possible the external world, the external reality.

And as I said before, the definition of reality is then confined to its physical description. And I think that when you look at the painting, for example, where there is an attempt to reproduce something about reality, the question is what the reality is in that painting? And what about the feelings that the painting can arise?

What about other sensations that the painting can give rise to? And these questions about subjective experiences, they’re so important. So I think that’s why a collaboration with the humanities is a great collaboration, although maybe some people think differently.

Well, Fulvio, your field, cognitive science, is a relatively new field, and I’m wondering, I mean, despite your characterization of your approach as dry, I’m wondering if, whether you think cognitive scientists in particular — and I realize there are many different forms of cognitive scientists — but whether you think cognitive scientists are especially open to learning from humanists or adopting multidisciplinary perspectives?

Well, I think they are by definition, in the sense that cognitive science has to do with the brain and the brain has to do with humans, for now. Of course we also, you know, apply some of the principles to study other living organisms who have brains more or less sophisticated than ours. But the very brain that we are studying is where the humanities come from, right? So ignoring that, ignoring that aspect, is a big mistake. It’s really a big mistake.

So I think that having, you know, a rigorous computational approach to study vision is okay, but that cannot be the only way of understanding this phenomenon, which, as I said before, has these very important subjective experiences, and we are still discussing if there is anything we have, any tool we have, that can actually characterize the subjective experiences.

I want to turn to a question that will start to draw in how the students contributed to the course. Massimo, you have mentioned a couple of times the fact that the course focused on vision, that vision was a key feature or the primary feature in thinking about the history of simulation. And in some ways, listening to you, Fulvio, there is an emphasis in your approach on illuminating how visual techniques are optimized and what ensures their success.

But of course, as we know, there is a history within the humanities which addresses the relation between vision and power, and which sees visual technologies in the service of forms of power, whether that be what feminist film critics have called the male gaze or what the social philosopher Michel Foucault described as disciplinary power through techniques of surveillance. And I’m just wondering, how … whether and how questions of power came up in the course, either in the topics addressed in the syllabus or from the side of the students. Massimo, maybe you could speak to that?

I’ll try. So, well, in the spirit of my project, in my lectures, I tried to stay as close as possible to representational case studies without superimposing also specific theoretical or ideological frameworks. So, but all the technologies we touched upon, and I mentioned a few earlier, are clearly intertwined with socioeconomic agendas, even racial dynamics, as famously exemplified — you mentioned Foucault — by Foucault’s interpretation of Bentham’s panopticon, which is … to many extents actualizes the centrality of vision in our culture, turning into a model, or the model, for a disciplinary and punitive power based on surveillance.

So there is no way to escape these questions. And I mentioned stereoscopic virtual travel kits along with panoramas. These virtual travel products played also a crucial role in the colonial stereotyping of cultures, including the Italian, as I show in my book. After all, “stereotype” and “stereoscope” have the same root. They come from “stereo” which in Greek means solid. So stereoscopes further solidified colonial and post-colonial stereotypes in many instances, and in their projects many of our students problematize the stereotypical effects of simulation.

Exactly a little bit of what Fulvio was mentioning before, how in an experiment we isolate certain components, and certain mechanisms of reality, and sort of ignoring all the contextual or even other concurrent dimensions of experience. And so I think in particular of a project by a Vietnamese student who looked at stereotypical representations of Vietnamese women, in particular in 19th-century French travel literature, including postcards and stereoscopic photographs, from both a historical and a techno-cultural perspective.

How this technology was used to reinforce or even to make even more tangible certain forms of representation, stereotypical forms of representation, and therefore functional to colonial power and to sort of a metropolitan understanding of the colonies. So another student has embedded a series of x-rays of her own hands into a suggestive multimedia piece of slam poetry focused on the question of visibility and her own racial identity.

Also reflecting on the meaning of dematerialization in contemporary digital culture. An extremely provocative project at the time when whole, less-visible communities were also suffering the brunt of the pandemic. So this was a project that, like many others — perhaps we mention them later — were able to make the most of the situation in which we found ourselves, mid semester, of being in a lockdown and having to run our experiments, thought experiments, etc. in the isolation of our bedrooms or studies or basements.

Those are wonderful examples. I’m curious, what sort of students did the course draw? Which concentrations or majors were represented among the students?

I think that overall the students were coming from the sciences, mostly from the sciences, which was actually something surprising, and for me, in a way, I was glad that that happened because there is more and more a kind of a running away of the sciences from the humanities, especially in recent times, and I think we should go back there, and I’m really convinced about that. And so these are Brown University students, so we have to be proud of them, and I think that’s why they were interested in it.

They were interested in these other aspects. And for me, what I have learned from this course, by listening to Massimo, and listening to these other perspectives, a way of thinking about vision information or in general information. So those were the kind of students, and there were also students from computer science and students who were actually interested in the technology of virtual reality, but yet they wanted to listen to the humanities point of view.

Were there some students from humanities departments in the course?

[Both answer, and Fulvio Domino says, “Sorry, Massimo, go ahead”]

There were a few, and I think MCM students, so students from Modern Culture and Media, so interested in the critical theory of media, etc. But the conversation was always very fluid and interesting. So these different points of view in the class the students never brought to a stalling point where, you know, you don’t know how to talk to each other, etc. I think Fulvio in his lectures in particular was able to, for humanities students in particular, to trigger their curiosity, their enthusiasm for the most technical aspects.

So I'm wondering, Massimo, were there any points of significant and unresolvable friction between the two approaches, the humanistic and the cognitive-scientistic, or were there ways in which it was impossible to integrate the two approaches in the course?

Well, this is an interesting question, Amanda. In fact I was somewhat concerned about potential friction between our points of view before the beginning of the seminar. But also I was eager to identify these potential points of friction because the most interesting questions for me fall precisely there, where science on the one hand and historical thinking might collide, presenting what seemed to be alternative or incompatible perspectives on what we call reality.

So namely I was eager to verify how historical and experimental perspectives may be productively combined. So whether understanding and describing how our perception works, the mechanisms of perception, of vision, and so on, that Fulvio studies, and the disembodied abstraction that sometimes characterizes a scientific approach, could help us better understand why, for instance, enhancing or altering our perceptions and playing with a technical simulation of our physical abilities and the world we live in has been such a constant in Western history and culture.

So why do we keep turning devices, born out of experiments designed to demonstrate how our perception works, such as the stereoscope, for example, into entertaining social games or gadgets capable to enhance or alter our perception of things? So, science alone cannot provide an intelligent answer to this question, and history alone cannot, or cultural history alone cannot explain it either without missing a crucial link between the reality of our physical bodies and the reality we as social beings construct or simulate around them.

So that’s how these points of friction became actually points of stimulation in the course. We tried to stimulate students to come up with experimental approaches that would both connect cultural or artistic experiences to scientific data and consider science and its applications as an answer, direct or indirect, to cultural challenges. So most of the projects the students turned in did, in creative and playful ways, confirm that there are no barriers between the scientific mind and cultural creativity as a whole.

The majority of students are used to experimental thinking since they came mostly from STEM or from the cognitive sciences. They did not have a problem to formulate cultural, historical, or political questions in terms of a controlled experiment. And students with a more humanistic background successfully blurred the boundaries between a thought experiment and an artistic installation.

That’s wonderful to hear you talk about that. Your course was, of course, interrupted by the COVID-19 pandemic, which forced a virtual experience on the course itself. I’m curious to hear, what was that experience like, and how would you characterize Zoom itself as a virtual space compared to some of your other objects of study? Fulvio?

Well, you have to look for a silver lining of the COVID-19 experience. I think that in this particular class it just came at the right moment. I mean, I’m not saying that anybody wanted that, but since it happened, we have to look at it in a positive way. And I think that what really happened is that after these few weeks of conversation about what is reality, what is simulated reality, what is the sense of immersion, and so forth, everybody had to go to their own place and be isolated, being in a way the least immersive possible in relationship with everybody else.

And I think that, you know, there could have been two kinds of reactions. One reaction is that everybody gets depressed and demotivated, and the other is that everybody gets stimulated. And what I believe is that perhaps the material covered in class before the appearance of the virus gave them this intellectual motivation to approach this real life challenge, and this was a test of their own conception of immersive experience.

So, all of a sudden, we have created through Zoom a new cyber-reality where everybody was together again in a way that transcended space and time. But to me, there was never a sense that the students went away. We were all in the same place as before, although this place was not located in a physical environment. So there you go, no better opportunity to ask the question of what is reality.

And Massimo? How is the experience of Zoom from your perspective?

It was actually positive. We were at the point where students were beginning to think about their own projects and the fact that we could have brainstorming sessions via Zoom, sharing on the screen drafts of their projects in progress and providing some suggestions. We met with them in small groups or individually with scheduled — you know, asynchronous outside of the class time, we scheduled these meetings, and they were very productive, and they helped us also keep in touch with the students and the students keep connected to the class.

And we tried to maximize this direct communication, you know, synchronous communication with them, because it was very important to maintain the cohesion of the class at that point, at the point where, you know, the collaborations were inevitably sacrificed due to that new situation.

What surprised you most in the experience of the course? I’m just curious to hear. Were there projects that the students did that really opened your eyes to particular aspects of the larger research collaboration that the course represented? Were there other things that happened over the course of the semester that surprised you?

To me the projects were just incredibly imaginative. So now the students had to go back to the drawing board because they could not work with physical stuff anymore or have their project appear in their common physical space. Instead, they were forced to do things in their own house, separated from everybody else, and then communicate their creation through Zoom.

But this almost dystopian scenario is what catalyzed a creative process with incredible results. To give you an example, a student from her own bedroom, in the house where she lives with her family, made a gigantic camera obscura, and the result was a stunningly surreal art installation. In the experiment done 1000 years ago by Ibn al-Haytham, a mathematician, astronomer, physicist, who created the first camera obscura, she blocked every source of light to her bedroom with the exception of a tiny pinhole in a window where she placed a small lens.

And now the external reality was only allowed to be presented to this very small aperture. And this created this magical, rarefied reality. There was an enchanting reproduction of what is out there and probably way more interesting to the senses than the crude physical reality where it was coming from. And then she actually used her family as human subjects to share their own experience. And the class, on the other hand, had their own experience of this through Zoom.

And what I’m pretty sure is that looking at the Zoom reproduction was different from her experience and that of her family. So in a way there were three interconnected realities that ended up to generate these three distinct realities. What an amazing way to reflect on what the meaning of reality is. And this is just one example that came to my mind of so many projects with similar stunning outcomes.

Would you like to add something, Massimo?

Yes. I mean, I totally agree that the students in fact were so resourceful and amazing. How they were able to connect these projects, conceived to some extent in abstract and in isolation, to their own emotional situation. I’m thinking of these two students who worked on a very, very interesting project based on photography, and how they interviewed a couple of artists, a Japanese artist, a woman artist, and a German artist, who superimpose photographs in real location in order to create this sense of a simulated reality, which becomes part of a real context.

Or in the case of the Japanese artist who, thanks to a very accurate technique using Photoshop and simulation technique, she basically inserted herself, herself as an adult, in her own picture, in the pictures of herself as a child. So creating a sort of time machine, the photograph as a time machine. And these students took the inspiration. They not only interviewed these artists, but also created her own project inspired by these artists.

And it was based on the Brown campus, you know, deserted by the COVID lockdown, and how they would go around the campus and, using the archive of old photographs of the Brown campus and Brown students at the Hay Library, they would superimpose different levels of reality, a photographic reality, including their own, her own.

They went around campus taking photographs of themselves in various locations and superimposing them with photographs, old photographs of other generations of students. So there was some sort of nostalgic, emotional quality to it that demonstrated an interesting point, but also in a very intimate way.

That’s a wonderful example. As a last question I would just like to know whether you have plans to teach the course again, and what you would do differently in a future iteration of the course?

Yes, we are going to teach this class again. It is scheduled for the next semester, the spring semester [Spring 2021], this time hopefully without having to give up the most interesting part, so, sacrificing again collaboration to confinement. So I look in particular forward to the opportunity that teaching this class in person may offer, provided we will be able to do it without jeopardizing anybody’s health. We will be finally able to work together in an actual lab.

I wish we are going to have a real vaccine, not a virtual vaccine, you know. And I mean, to really see the students and interact together. That’s when reality, the physical reality comes back full force.

It’s been an absolute pleasure speaking with you both. Thanks so much for being on the show.

Thank you.

Thank you.

Thanks for listening to this episode of Meeting Street. You can find a video with selected student projects from Fulvio and Massimo’s course on our podcast page, along with the show’s transcript. A special note on the last project described by Massimo: The students’ idea of superimposing older pictures on the deserted Brown University campus was inspired by the work of Providence photographer Mike Cohea, who did a similar project using photographs from the Brown University archives. Cohea’s project, “Brown: Then, Now & Forever,” can be viewed at MikeCohea.Com or on the Brown University Facebook page.

The other photography projects presented by the students, and mentioned by Massimo during the interview, were Chino Otsuka’s “Imagine Finding Me” and Michael Hughes’s “Souvenirs.” We hope you enjoyed hearing about this innovative teaching collaboration between a humanities scholar and a cognitive scientist. Please join us again for our next show.